1 Stimuli

1.1 Summary

Participants in the ENG-SPA listened to 103 Spanish words, and participants in the ENG-CAT and the SPA-CAT groups listened to 86 trials. Five trials in each condition were practice trials and were excluded from analyses.

Table 1
Lexical frequency, phonological neigbourhood size, and levenshtein distance of the stimuli words.
Freq. (million) Freq. (Zipf) PTHN Levenshtein
Median SD Median SD Median SD Median SD
ENG-CAT 19.89 69.01 4.34 0.55 2.00 4.39 6.25% 18.71%
ENG-SPA 23.67 65.95 4.37 0.55 2.00 4.19 0.00% 17.15%
SPA-CAT 16.15 155.34 4.23 0.62 0.50 2.61 33.33% 25.50%
Median 19.89 — 4.34 — 2.00 — 6.25% 18.71%
SD 3.76 — 0.08 — 0.87 — 17.72% 4.44%
Note. Median and standard deviation (SD) of lexical frequencies of target words (expressed in counts per million and as Zipf scores), density of higher-frequency phonological neighbourhoods of target words, and normalised Levenshtein phonological similarity between the presented and the target words.

1.2 Lexical frequency

1.3 PTHN

1.4 Levenshtein distance

1.5 Distributions

Distribution and summary statistics of stimuli translations, plotted in different panels for English translations of Catalan stimuli (cat-ENG), English translations of Spanish stimuli (spa-ENG), and Spanish translation of Catalan stimuli (cat-SPA). A\) Lexical frequency of stimuli translations, expressed in counts per million and as Zipf scores. B\) Number of higher-frequency phonological neighbourhoods (PTHN) of stimuli translations. C\) Normalised Levenshtein phonological similarity between the presented stimuli and their translations, expressed in percentages. Lexical frequencies and PTHN scores of the stimuli (presented auditorily in Catalan or Spanish) are not described, as participants lack any lexical representation of such words.

Distribution and summary statistics of stimuli translations, plotted in different panels for English translations of Catalan stimuli (cat-ENG), English translations of Spanish stimuli (spa-ENG), and Spanish translation of Catalan stimuli (cat-SPA). A) Lexical frequency of stimuli translations, expressed in counts per million and as Zipf scores. B) Number of higher-frequency phonological neighbourhoods (PTHN) of stimuli translations. C) Normalised Levenshtein phonological similarity between the presented stimuli and their translations, expressed in percentages. Lexical frequencies and PTHN scores of the stimuli (presented auditorily in Catalan or Spanish) are not described, as participants lack any lexical representation of such words.

2 Participants

We collected data from June 04th 2020 to June 28th 2020. We tested 104 participants. 36 were English natives tested in Spanish, 37 were English natives tested in Catalan, and 32 were Spanish natives tested in Catalan. 14 failed to meet all the inclusion criteria and were excluded from further analyses. The final sample comprised data from 14 participants, 28 in the ENG-SPA group, 34 in the ENG-CAT group, and 29 in the SPA-CAT group.

Participants were included if1:

  • Aged 18 to 25 years
  • Did not report being fluent in Catalan, Spanish, or Italian
  • Provided at least 80% of the presented trials
  • Did not report motor, auditory or visual (other than wearing glasses) problems

Trials were considered valid if:

  • Participant did not leave a blank response
  • Participant took longer than 10 seconds to respond

2.1 Summary

2.2 Valid trials

2.3 Demographic information

2.4 Second language

2.5 Language profile proficiency

3 Data analysis

We fitted a Bayesian multilevel regression model using the R package brms (Bürkner, 2017), with correct responses (correct, 0 is incorrect response, 1 if correct response) as the response variable with a Bernoulli distribution and a logit link function. We modelled the probability of an average participant providing a correct response, conditional to a set of predictors (aka. fixed effects) and their interactions:

  • PTHN (pthn, 0-Inf): number of phonological neighbours with higher frequency of the target word. Extracted from the CLEARPOND database. Standardised before entering the regression model.
  • Phonological similarity (lv, 0-1): Inverse Levenshtein distance between the IPA transcription of the presented word and its correct translation in the target language, calculated using the stringdist function of the stringdist package (van der Loo, 2014). Standardised before entering the regression model.
  • Shared phonological onset (onset, Same/Different): Whether the presented and the target word share phonological onset, as judged by experimenters, and sum coded as \(Different = -0.5\) and \(Same = +0.5\).

We added participants (participants) and as a grouping variable (aka., random effects), therefore adding participant-level adjustments to the model. We specified random intercepts slopes for all fixed effects within the participant grouping variable.

We performed multiple imputation via predictive mean matching using the mice function of the mice package (Van Buuren & Groothuis-Oudshoorn, 2011) to impute missing values in the response variable or the predictors.

The model was implemented using the following formula (fit_3):

$$\[\begin{align} &\textbf{Likelihood} \\ y_{i} \sim& Bernoulli(p_{i}) && \text{[probability of correct translation]} \\ \\ &\textbf{Parameters} \\ logit(p_{i}) = ~ & \beta_{0[p,w]} ~ + && \text{[linear model]}\\ & \beta_{1[p]} ~ Frequency_{i} ~ + \\ & \beta_{2[p]} ~ PTHN_i ~ + \\ & \beta_{3[p]} ~ Similarity_i ~ + \\ & \beta_{4[p]} ~ (PTHN_i \times Similarity_i) \\ \\ \beta_{0-6[p,w]} \sim& ~ \mathcal{N}(\mu_{\beta_{j}}, \sigma_{\beta_{j}}) \text{, for participant } p ~\text{in 1, ..., } P ~\text{and word } w ~\text{in 1, ..., } W && \text{[participant- and word-level intercepts]} \\ \beta_{1-6[p]} \sim& ~ \mathcal{N}(\mu_{\beta_{j}}, \sigma_{\beta_{j}}) \text{, for participant } p ~\text{in 1, ..., } P && \text{[participant-level coefficients]} \\ \\ &\textbf{Prior} \\ \mu_{\beta_{p,w}} ~ \sim& ~ \mathcal{N}(0, 0.1) && \text{[participant-level coefficients]} \\ \sigma_{\beta_{p}}, ~ \sigma_{\beta_{w}} \sim& ~ HalfCauchy(0, 0.1) && \text{[SD for population and participant]} \\ \rho_{p}, ~ \rho_{w} \sim& ~LKJ(8) && \text{[correlation between participant-level coefficients]} \\ \end{align}\]$$

The estimation of the model was performed using Bayesian inference via Hamiltonian Monte-Carlo in Stan (Carpenter et al., 2017), with 4 chains, 2,000 iterations each (1,000 warm-ups iterations). The output of this model is the approximated posterior distribution of all the parameters estimated, therefore indicating the probability of each combination of values of the parameters, conditional to the observed data.

We compared this (extended model) to other models dropping one predictor at a time using Leave-one-out cross-validation (LOO-CV), using the loo and loo_compare functions of the brms package. We compared our extended model against the following models:

  • fit_0: ~ 1 + (1 | participant) + (1 | word)
  • fit_1: ~ 1 + frequency_zipf + (1 + frequency_zipf | participant) + (1 | word)
  • fit_2: ~ 1 + frequency_zipf + pthn + (1 + frequency_zipf + pthn | participant) + (1 | word)
  • fit_3: ~ 1 + frequency_zipf + pthn + lv + (1 + frequency_zipf + pthn + lv | participant) + (1 | word)
  • fit_4: ~ 1 + frequency_zipf + pthn + lv + pthn:lv + (1 + frequency_zipf + pthn + pthn:lv + lv | participant) + (1 | word)
  • fit_5: ~ 1 + frequency_zipf + pthn + lv + pthn:lv + group + (1 + frequency_zipf + pthn + pthn:lv + lv | participant) + (1 | word)
  • fit_6: ~ 1 + frequency_zipf + pthn + lv + pthn:lv + group + group:lv + (1 + frequency_zipf + pthn + pthn:lv + lv | participant) + (1 | word)

4 Results

4.1 Model comparison

Leave-one-out cross-validation. The more negative is elpd_diff (and the larger its magnitude compared to its corresponding standard error), the better it first the data compared to the null model (~ 1 + (1 | participant)). The model that fits the data the best is the extended model.

Table 2
Model performance and comparison using leave-one-out cross-validation
LOOELPD SE LOOIC SEIC LOOdiff SEdiff
Model 6 −1,946.90 47.56 3,893.80 95.13 - -
Model 3 −1,948.45 47.17 3,896.90 94.34 −1.55 3.05
Model 5 −1,950.08 47.14 3,900.15 94.28 −3.18 2.85
Model 4 −1,951.06 47.26 3,902.13 94.53 −4.17 2.91
Model 2 −2,030.32 48.90 4,060.64 97.80 −83.42 15.11
Model 1 −2,033.20 48.87 4,066.39 97.75 −86.30 15.35
Model 0 −2,053.34 49.27 4,106.68 98.54 −106.44 16.43
Note. Outcomes of the leave-one-out cross-validation procedure. Each row indicates the values of the expected log-predicted density (ELPD), leave-one-out information creterion (LOO-IC), and the difference in ELPD (LOO-diff) between each model and the inmediately simpler model. Each value is accompanied by a standard error (SE) indicating the uncertainty of its estimate.

4.1.1 Fixed coefficients

Predictor Mean 95% CrI
Intercept 43.08% [38.20%, 48.00%]
Frequency [+1 SD] 1.14% [−2.40%, 4.92%]
PTHN [+1 SD] −1.70% [−5.26%, 1.98%]
Levenshtein [+1 SD] 12.30% [7.48%, 17.39%]
NA 3.74% [−1.09%, 8.64%]
NA 0.60% [−3.98%, 5.17%]
PTHN × Levenshtein −0.11% [−3.72%, 3.54%]
NA 3.42% [−1.52%, 8.45%]
NA 3.08% [−1.24%, 7.43%]
Estimated posterior distributions of coefficients in Model 6. A\) Population-level effects. Distributions indicate the estimated posterior likelihood density of regression coefficients of fixed effects. Credible intervals (*CrI*), represented with increasingly lighter segmentents in the distribution indicate the range of values that contain the true value with 95\%, 80\%, and 50\% probability. Black dots represent the mean of each distribution.
Estimated posterior distributions of coefficients in Model 4. A\) Population-level effects. Distributions indicate the estimated posterior likelihood density of regression coefficients of fixed effects. Credible intervals (*CrI*), represented with increasingly lighter segmentents in the distribution indicate the range of values that contain the true value with 95\%, 80\%, and 50\% probability. Black dots represent the mean of each distribution. B\) Participant\-level coefficient variability. The model estimated participant\-level coefficients to account for the dependency between responses from the same participant. Distributions in this panel indicate the estimated variability across coefficients from different participants, expressed as standard deviations (*SD*). C\) Correlation between participant\-level effects. The model allowed participant-level coefficients to co\-vary. This panel represents the Pearson correlations between each pair of coefficients, expressed as the mean of the posterior distribution of each estimated correlation.

Estimated posterior distributions of coefficients in Model 4. A) Population-level effects. Distributions indicate the estimated posterior likelihood density of regression coefficients of fixed effects. Credible intervals (CrI), represented with increasingly lighter segmentents in the distribution indicate the range of values that contain the true value with 95%, 80%, and 50% probability. Black dots represent the mean of each distribution. B) Participant-level coefficient variability. The model estimated participant-level coefficients to account for the dependency between responses from the same participant. Distributions in this panel indicate the estimated variability across coefficients from different participants, expressed as standard deviations (SD). C) Correlation between participant-level effects. The model allowed participant-level coefficients to co-vary. This panel represents the Pearson correlations between each pair of coefficients, expressed as the mean of the posterior distribution of each estimated correlation.

4.2 Random effects

4.3 Marginal effects

Expected mean posterior predictions from Model 4. A) Population-level predictions: the X-axis and the Y-axis represent the PTHN score (in standard deviations from the mean) and the probability of correct translation, respectively. We simulated 300 observations from the posterior distribution of the model: 100 simulations for translations with no similarity (0% Levenshtein), 100 simulations for translations with mean accuracy (20.51% Levenshtein), and 100 simulations for translations with the maximum observed accuracy (83.33% Levenshtein). We did this across the range of values of the PTHN scores. For each simulation, we drew a single sample from the posterior distribution of each coefficient. Each simulation is depicted in the graph as a line: pink for minimum similarity translations, blue for hight similarity translations, and yellow for maximum similarity translations. Black, thick lines indicate the expected mean value of the posterior predictions of the model for each condition. The dispersion of the lines indicates the uncertainty of our predictions. We computed these posterior predictions for each group of participants, and plot tem in separate panels.

Expected mean posterior predictions from Model 4. A) Population-level predictions: the X-axis and the Y-axis represent the PTHN score (in standard deviations from the mean) and the probability of correct translation, respectively. We simulated 300 observations from the posterior distribution of the model: 100 simulations for translations with no similarity (0% Levenshtein), 100 simulations for translations with mean accuracy (20.51% Levenshtein), and 100 simulations for translations with the maximum observed accuracy (83.33% Levenshtein). We did this across the range of values of the PTHN scores. For each simulation, we drew a single sample from the posterior distribution of each coefficient. Each simulation is depicted in the graph as a line: pink for minimum similarity translations, blue for hight similarity translations, and yellow for maximum similarity translations. Black, thick lines indicate the expected mean value of the posterior predictions of the model for each condition. The dispersion of the lines indicates the uncertainty of our predictions. We computed these posterior predictions for each group of participants, and plot tem in separate panels.

Bürkner, P.-C. (2017). Brms: An r package for bayesian multilevel models using stan. Journal of Statistical Software, 80(1), 1–28.
Carpenter, B., Gelman, A., Hoffman, M. D., Lee, D., Goodrich, B., Betancourt, M., Brubaker, M. A., Guo, J., Li, P., & Riddell, A. (2017). Stan: A probabilistic programming language. Grantee Submission, 76(1), 1–32.
Van Buuren, S., & Groothuis-Oudshoorn, K. (2011). Mice: Multivariate imputation by chained equations in r. Journal of Statistical Software, 45(1), 1–67.
van der Loo, M. P. J. (2014). The stringdist package for approximate string matching. The R Journal, 6, 111–122. https://CRAN.R-project.org/package=stringdist

  1. We originally planned to exclude participants that reported any visual impairment that glasses would not correct. This item was phrased as Do you have normal or corrected-to-normal VISION? (Yes/No) in English, and as ¿Tienes problemas de VISIÓN que unas gafas o lentes de contacto NO corrijan? (Sí/No). Surprisingly, the proportion of Spanish participants that reported visual impairment was implausibly large (n = 6, 18.18%). This is possibly due to some participants using glasses daily and not having read the item until the end, where it is indicated that the use of glasses is considered as normal vision↩︎